Performance enhancing proxy techniques for internet protocol traffic

ABSTRACT

Methods and systems are described that may be used in the context of a Performance Enhancing Proxy architecture for Internet traffic. A method of accelerated connection opening with error handling is disclosed. Also, a method of handling the Path MTU Discovery mechanism in the context of a distributed connection splitting PEP is described. A novel packet format for a proprietary inter-PEP protocol is described which allows for low packet header overhead. A new acknowledgement scheme that adapts to packet loss conditions to minimize bandwidth consumption by selecting an acknowledgement type from several possibilities is also detailed. A method whereby potentially spurious retransmissions are minimized by timing every transmission and retransmission is also described.

CROSS-REFERENCE TO RELATED APPLICTIONS

[0001] This application claims priority to U.S. Provisional ApplicationNo. 60/333,608 to Jason D. Neale et. al., entitled “PerformanceEnhancing Proxies for Satellite Transmission Control Protocols,” filedon Nov. 13, 2001.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The invention relates to the field of multiplex communicationstechniques. More particularly, the invention provides methods andsystems for improving the performance, efficiency and user experience ofsystems transporting Internet Protocol (IP) traffic, by the use ofPerformance Enhancing Proxies (PEPs).

[0004] 2. Background of the Invention

[0005] The Internet is a world-wide computer super-network, which ismade up of a large number of component networks and theirinterconnections. Computer networks may consist of a wide variety ofconnected paths or network links serving to transport user informationin the form of data between a diverse array of computer end systems.Different network links are more or less suitable for different networkrequirements. For example, a fiber optic cable typically provides a highbandwidth, low per bit cost, low error rate and low delay point-to-pointnetwork link. Alternatively, for example, a satellite link typicallyprovides a lower bandwidth, higher per bit cost, higher error rate andlonger delay point-to-multi-point network link. The wide variety oflinks and thus link characteristics encountered on the Internet, orother private (IP) based networks, have a variety of effects on thebehavior of protocols in the IP suite.

[0006] IP primarily provides the routing functionality for packets (bitsor bytes of data) over a network. It acts at the network layer to directpackets from their sources to their destinations. Transmission ControlProtocol (TCP) is the reliable transport layer protocol of the IP suiteof protocols and as such, layers on top of IP, provides reliability toapplications, and builds on IP's unreliable datagram (packet) service.TCP underlies the vast majority, estimated to be around 90%, of all thetraffic on the Internet. TCP supports the World Wide Web (WWW),electronic mail (email) and file transfers, along with other commonapplications. TCP was introduced in 1981 and since then has evolved inmany ways, but today still provides reliable and largely efficientservice over a wide variety of links as evidenced by its omnipresentnature. However, there are a variety of conditions under which TCP mayperform below expectations, geosynchronous satellite links being oneprime example. The problems of TCP over satellites has been previouslydocumented. TCP performance is typically degraded to some extent interms of lowered throughput and link utilization by, but not limited to,the following link characteristics: long delay, high bandwidth, higherror rate, link asymmetry and link variability, all of which may beencountered on satellite and similar links.

[0007] In response to the established use of TCP and also of certainlink types (such as satellite) which are not ideal for TCP, PerformanceEnhancing Proxies (PEPs) have been introduced. PEPs may function as oneor more devices or pieces of software placed in the end-to-end path thatsuffers TCP performance degradation. PEP units may, for example,surround a satellite link. PEPs modify the traffic flow in an attempt toalleviate the issues of TCP traffic on a specific link. PEPs may usemany methods, either alone or in concert, to enhance performance.

[0008] One type of PEP, known as a distributed, connection splitting PEPis commonly chosen due to that fact that it allows for the use of aproprietary protocol across the satellite link. This protocol can thenbe chosen or designed to mitigate problems specific to the link. Adistributed PEP uses more than one PEP device in an end-to-endconnection, often two PEP devices are used. If two PEP devices are used,the end-to-end connection may be split into 3 connection segments. Theend connections must remain TCP for compatibility, but the inter-PEPconnection may be any protocol. Several protocols are available for useon the satellite link that provide improved performance over that ofTCP. Examples of these protocols are Xpress Transport Protocol (XTP),Satellite Transport Protocol (STP), Space Communications ProtocolStandards—Transport Protocol (SCPS-TP), standard or enhanced UserDatagram Protocol (UDP) or even non-standard (modified) TCP. In additionto the protocol used, there are also many ways in which a PEP device mayhandle processing between connection segments in this type of system.

[0009] One of the link characteristics that affects TCP performance isdelay. Links such as those over GEO satellites have long delays, forexample, around 500 ms or more. Several TCP mechanisms that controlconnection setup, flow control and error correction throughretransmission may be adversely affected by long delay links.

[0010] For transfers that are typically short in duration, such as webpages, the delays involved in establishing TCP connections make aproportionally larger contribution to the transfer time, and thereforeto the mean data throughput rate. Additionally, a user typically beginsto view a web page as it is downloading so an initial delay before anymaterial is displayed may frustrate a user and also consequently,potentially cause re-requests which lower system efficiency.

[0011] The delay in connection opening is caused by a mechanism known asthe TCP Three Way Handshake (3WHS). The purpose of this exchange ofmessages is to ensure that the other end point is present, and therebyto promote the reliability of a transfer. A connection initiator sends apacket with the SYN (synchronize) flag set. A responding system sendsback a packet with the SYN and ACK (acknowledgement) flags set. The ACKflag acknowledges the initiator's SYN. The initiator then sends a finalACK packet acknowledging the responder's SYN. From this point on theinitiator may send data. Thus, the delay from initiation to sending dataon a TCP connection is a whole RTT [Note to inventor—please define RTT].

[0012] When opening a TCP connection in a distributed split-connectionPEP implementation, there are two main options and then variantsthereof. For preserving end-to-end behavior of the connection andreliability, the connection should be opened end-to-end and theconnection should be opened by the endpoints and not the PEP devices.Although more reliable than alternatives, however, this method involvesa full RTT of overhead during which no data is transferred. Analternative method involves accelerating the opening of certainconnections, such as web connections, which are of short duration andthus more heavily affected by extra RTTs.

[0013] An initiator sends a SYN packet to a PEP and the PEP respondslocally with the SYN/ACK packet to the initiator. The initiator thenresponds with the ACK packet and the first data packet, which in thecase of a web transfer is an HTTP request packet. The PEP then combinesthe original SYN packet (which it has held) and the first data packetand sends them over the satellite link to the other PEP device. Thelower RTT on the terrestrial link means that the time taken to send thefirst request is reduced.

[0014] A problem with the above accelerated opening is that it ispossible to open a connection locally that might then fail to establishend-to-end, resulting in a desynchronized state. This state willeventually time-out. However, during the time that the two endpoints aredesynchronized, the user will be confused, as the connection appears tobe established but no data will be transferred, which again could leadto the user re-attempting the connection several times and wastingbandwidth.

[0015] As described earlier, the Internet is a collection of networksand interconnections. These interconnections and network links each havetheir own characteristics. One characteristic is the MaximumTransmission Unit (MTU) size. This value, often expressed in bytes, isthe maximum data payload that may be encapsulated and carried over theOSI/ISO 7-layer model link layer without being broken down into asmaller unit. Two common technologies for LAN links are Ethernet and thesimilar, but not identical, IEEE 802.3 standards. Ethernet allows forthe encapsulation of a 1500 byte IP packet (1500 byte MTU) while 802.3encapsulation allows for a 1492 byte MTU. It can be imagined that in anetwork of heterogeneous links there will, sometimes, not be one commonMTU for any path between points A and B in a given network or paththrough the Internet.

[0016] In response to the recognition that any given path through anetwork may not have a consistent MTU for all hops, the IP protocolallows for fragmentation of IP packets. If the IP layer at a host orrouter is unable to send a packet of the desired size onto the link, theIP layer will split that packet up into several smaller packets. Whenthis behavior occurs at a router between ports, it is known asfragmentation and is commonly recognized to have detrimental sideeffects, such as lowering maximum data rate (through additional headerbytes and also packet processing overhead at network nodes) andimpacting efficiency. However, fragmentation is necessary to allow thedata to pass end-to-end.

[0017] In an attempt to avoid fragmentation, the process of Path MTUDiscovery (PMTUD) was introduced. The purpose of this process is to tryto detect the minimum MTU in the path from source to destination. Thisvalue is dynamic if the route changes. The IP header has a flag, whichmay be set to inform intermediate network nodes (i.e. any devices in thenetwork between the source and destination) not to fragment a packet.This flag is known as the Don't Fragment (DF) flag. When the DF flag isset, a router should discard the packet if it is too large to forward onthe outgoing interface. The router should also send an Internet ControlMessage Protocol (ICMP) Can't Fragment (ICMP type 3 [destinationunreachable], code 4 [fragmentation needed but don't-fragment bit set])message back to the originator of the packet. This packet should containthe MTU of the outgoing interface on the router to inform the sender ofthe limiting MTU. Through this mechanism, a sender may adapt to the pathMTU and avoid fragmentation. This mechanism is therefore desirable forefficiency reasons.

[0018] Currently, there is little guidance on how PMTUD should functionin the presence of PEPs. In the absence of guidance, it is currentlyleft to the decision of each PEP designer or manufacturer on how tohandle the PMTUD mechanism at a PEP. One solution requires that ICMPmessages pass through its PEP devices without modification. This allowsfor the sender to adapt its path MTU estimate and send smaller packetsin the future.

[0019] However, a problem exists in a connection-splitting distributedPEP, due to the fact that the PEP devices are often buffering packetsthat are in transit between the endpoints. These packets have beenacknowledged to the sending endpoint and are, therefore, no longerbuffered by the endpoint itself for retransmission. Therefore, if arouter drops a packet after the second PEP in the connection and an ICMPCan't Fragment message is sent to the originator, a problem occurs. Theoriginator is able to lower the Path MTU estimate but cannot retransmitthe data in the original packet. The second PEP in the connection has acopy of the packet buffered so may retransmit when no TCPacknowledgement arrives, but will not understand that the packet must beresized to a smaller packet to arrive successfully at the destination.Therefore, a deadlock may occur until several retransmissions of thepacket have failed and the connection has to be reset.

[0020] One solution to this problem is that PMTUD may be disabled when aPEP is included in the end-to-end connection to allow the connectionsusing the PEP to function correctly. This however is not ideal for thereasons stated above. Hence, problems exist in the current technique forPMTUD when PEPs are used.

[0021] Each protocol used on the Internet has its own packet format,which specifies the way that information is encoded in headers and wheredata begins in a packet, among other things. The TCP packet formatincludes the TCP header and space in the header for optional fieldsknown as TCP options. Distributed connection splitting PEPs may useother (non-TCP) standard protocols and possibly proprietary protocolsbetween the two PEP devices. These non-TCP protocols are used to gainperformance advantages over end-to-end TCP and even split connectionTCP, performance however is only one, although the most important,aspect of a PEP. A PEP must also be compatible with the end hosts andthe TCP protocol. If the PEP to PEP protocol does not support thetransfer of certain TCP information from end-to-end then functionalitywill be lost; the TCP urgent pointer which is used to expedite transferof portions of the data stream being one example.

[0022] When choosing or designing a protocol for the problematic linkthere is, therefore, a tradeoff between efficiency and compatibility. Ifusing an entirely different protocol, it may be necessary to carry theTCP information in extra header structures, which may increase thepacket overhead on each packet. Increasing packet overhead may alsotrigger IP fragmentation for packets that were originally the maximumsize for the link; this should be avoided. Also, the end-to-end pathover which the connection travels may have intermediate equipment thatdoes not know how to handle unknown protocols. For example, NetworkAddress Translation (NAT) devices may perform translation of the IPaddress fields and sometimes layer 4 protocol port numbers also. Thesetypes of operations can then require the checksum fields to be updated.If a protocol is not recognized, it may not be able to function properlyat, for example, the NAT device or packets may pass the NAT device butbe unrecognizable at the receiver. Additionally, the functionality of anewly designed protocol will impose constraints on the information thatmust be carried in each packet. For the proprietary protocol chosen foruse with the PEP design of this invention, no pre-existing packetstructure was considered appropriate.

[0023] For problematic links, TCP has been improved by several differentmechanisms to address different issues. For the case of packet andacknowledgement loss, TCP has been improved by the addition of theSelective Acknowledgement (SACK) option. This allows TCP packet headersto carry information on contiguous blocks of packets that have beensuccessfully received. This mechanism adds overhead to each packet andalthough the overhead is only a small percentage on large packets(around 1% on a 1500 byte packet), the percentage overhead on a standardacknowledgement packet is much larger. For a 40-byte packet, an extra 12to 20 bytes of SACK information is between an extra 30 and 50% of theoriginal packet size. More seriously, if the TCP acknowledgements arecarried over a link layer protocol such as Asynchronous Transfer Mode(ATM), a TCP acknowledgement with SACK information may no longer fitwithin a single ATM cell. If, instead, two cells are required for theacknowledgement then acknowledgement traffic volume is, in effect,doubled. If, for example, this is the return channel on a satellitesystem such as the Digitial Video Broadcast-Return Channel Satellite(DVB-RCS) where most traffic may be acknowledgement traffic, then thetotal traffic volume may also be nearly doubled.

[0024] TCP also uses a cumulative acknowledgement scheme to signalcorrect reception of packets to the sender. Optionally, TCP may use theSACK option described earlier if higher packet loss rates are expected,as may often be the case over satellite links, for example. Whetherstandard TCP acknowledgements are used or whether the SACK option isused, the same method of acknowledgement must be used throughout theduration of the connection. If the error conditions on the link changeduring the course of the transfer, the connection performance may beadversely impacted if an inappropriate acknowledgement method is chosen.For example, if the standard TCP acknowledgement scheme is selected, theTCP transfer may suffer very poor performance or even failure underheavy error conditions. If the SACK scheme is chosen, the additionaloverhead, as described above, may be incurred even if the SACK scheme isnot needed. TCP is unable to adapt the acknowledgement scheme tochanging error conditions during the course of the connection. Thisproblem exists with the conventional systems in the area ofacknowledgement of packets.

[0025] TCP also uses a timer as one method of detecting lost packets andtriggering retransmissions. However, in the conventional systems, onlyone timer is used regardless of how many packets are being sent. TCPuses the timer in the following manner. When there are no packets intransit, the timer is off. When the first packet is transmitted, thetimer is set. When a packet is acknowledged and other packets are stillin transit, the timer is reset. Therefore it may take different amountsof time to detect a packet loss depending upon which packet in a groupis lost. In the worst case it may take up to the timer timeout valueplus the round trip time to detect a loss. This time period may bealmost twice as long as the detection period for loss of the firstpacket. In the ideal case, every loss should be detected as quickly aspossible.

[0026] Additionally, and perhaps more importantly, if an acknowledgementscheme is used in which repeated retransmission triggers occur for thesame packet, the single packet timer provides no indication of how longan individual packet has been in transit. This means that it is notpossible to know if a transmitted packet has had time to be acknowledgedor not. In this case, it is possible to retransmit a packet before ithas bad time to reach the destination and an acknowledgement be returnedand received. This scenario lowers the efficiency of the link as packetsare transmitted multiple times unnecessarily. This is a problem in theconventional systems related to controlling or limiting unnecessaryretransmissions.

SUMMARY OF THE INVENTION

[0027] The invention provides solutions to the problems of theconventional systems as described above in the following areas:connection opening, path MTU discovery, satellite protocol packetformat, acknowledgement of satellite protocol packets and unnecessaryre-transmissions. The invention provides systems and methods forconnection opening in a distributed split-connection PEP implementation.The systems and methods in accordance with the invention use anaccelerated opening that allows for minimizing the time spent in thedesynchronized state to avoid unnecessary re-request transfers in thecase where the connection does not establish properly. When a connectionattempt fails, the current invention intercepts the commonly resultingICMP messages at the PEP, and handles them to provide a faster tear downof the failed connection. This minimizes the time the endpoints spend ina desynchronized state and reduces the period in which a user may becomefrustrated and generate multiple re-requests. Thus, one aspect of theinvention is to reduce unnecessary request transmissions, for examplefor web pages, and to improve the user experience by avoiding long idleor dead periods which cause the connection to seem to have stoppedresponding.

[0028] The invention also provides a solution to the problems regardingthe use of the PMTUD mechanisms in the presence of PEPs, specificallywhere the PEPs locally acknowledge packets and buffer them forretransmission. The invention offers a system and method where the PEPsinvolved in an end to end connection filter for ICMP “Can't Fragment”messages for the PEP enhanced TCP connections “If Can't Fragment”messages are received then the path MTU estimate at the PEP is adjustedand the data contained in the dropped packet is locally retransmitted inmultiple subsequent packets. This avoids a potential deadlock that mayprove fatal to affected connections. Therefore, a further aspect of theinvention is to provide a reduction of unnecessary processing andretransmissions from the PEP devices.

[0029] The invention also provides a novel packet structure to supportthe unique needs and functionality of a proprietary protocol designed tomitigate certain detrimental link characteristics. The packet structurein accordance with the invention is compact, being the same size as theminimal TCP header, which also means that fragmentation is avoidedbetween the PEP devices. The inventive packet structure supports some ofthe TCP functionality and also new features of the proprietary protocol.The protocol uses the TCP protocol type in the IP header so that it willbe treated like TCP by intermediate equipment such as routers and NATs.

[0030] The port numbers are not altered, as they are used by both TCPand the proprietary protocol for connection identification. The portnumbers may be modified by a NAT device if necessary, as they arelocated at the same place in the packet structure. The TCP checksum isalso used, again for compatibility, primarily with NAT devices. Sequenceand acknowledgement numbers are modified to allow for behaviourdifferent to TCP. The TCP flags are shared by both the TCP endconnections and the proprietary protocol. The TCP reserved field ismaintained for compatibility with future uses. The TCP window size fieldis reused for communicating a satellite protocol PEP-to-PEP flow controlwindow. The urgent pointer is maintained.

[0031] A packet number field which is 24-bits in length replaces the TCPsequence number field. An acknowledgement number field is also includedwhich is also 24-bits in length. The additional spared bits from thepacket and acknowledgement fields are used for identifyingacknowledgement type, and also as bit flags to represent packets beingacknowledged. Each bit flag indicates the presence or absence of apacket at the receiver. Depending on the value of the TCPacknowledgement flag and the acknowledgement type flag, differentacknowledgement formats may be used. The acknowledgement scheme alsoallows for sending multiple acknowledgements in one packet, without theneed for a larger packet header, through the use of options.

[0032] The invention also offers a system and method whereby aproprietary protocol may use several different types of acknowledgementpackets. The packet types allow for positively, and possibly negatively,acknowledging a different number and pattern of received and possiblymissing packets. Additionally, a method for choosing the most suitableacknowledgement to use each time an acknowledgement is sent isdescribed, with the intention of reducing the volume of bytestransmitted to acknowledge a pattern of received and lost or erroredpackets. This scheme allows for the possibility of dynamically adaptingto packet loss conditions to lower the volume of acknowledgement bytesotherwise necessary. The scheme also has lower overhead than schemes inprotocols such as TCP. The inventive scheme may also employ anacknowledgement timer to ensure a minimum rate of acknowledgements andan upper bound to the real RTT.

[0033] The invention also allows for the timing of every packet intransit. Each time a packet is transmitted, a copy is buffered to allowfor retransmission if necessary. Each buffered copy of a packet has,stored with it, a timestamp that records the time of transmission.Alternatively, the buffered copy of the packet may have stored with it atimestamp indicating the expected time of acknowledgement. In the firstscheme, the stored timestamp may be compared against the current time tosee if the time difference is greater than the expected roundtrip time,including all delays. In the second scheme, the current time is comparedagainst the stored time to see if an acknowledgement for the packet wasexpected by this time. Both methods, therefore, can be used to preventre-transmitting a packet if a copy of the packet is already in transiton the link or an acknowledgement is in transit in the return direction.The units of time used may be real or pseudo time units. When packetsare retransmitted the timestamps are updated.

BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are included to provide afurther understanding of the invention and are incorporated in andconstitute a part of this specification, illustrate embodiments of theinvention. Together with the written description, these drawings serveto explain the principles of the invention. In the drawings:

[0035]FIG. 1 shows a block diagram of an exemplary PEP deployment withthe connections and equipment involved;

[0036]FIG. 2 shows a split-connection, distributed PEP implementation inaccordance with an embodiment of the invention;

[0037]FIG. 3 illustrates the system for reducing desynchronised time byhandling ICMP messages at the PEPs in accordance with an embodiment ofthe invention;

[0038]FIG. 4 shows PEP intervention in the Path MTU Discovery mechanismin accordance with an embodiment of the invention; and

[0039]FIG. 5 shows the proprietary protocol packet format in accordancewith an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0040] This application incorporates by reference the disclosures ofprovisional patent application No. 60/333,608, entitled “PerformanceEnhancing Proxies For Satellite Transmission Control” from which thisapplication claims priority, an application entitled “Flow ControlBetween Performance Enhancing Proxies Over Variable Bandwidth SplitLinks” and an application entitled “Performance Enhancing Proxies” bothof which are concurrently filed herewith on Nov. 13, 2002 and havecommon ownership.

[0041]FIG. 1 illustrates a simplified view of a system 100 that includesequipment and links involved in a PEP deployment in a satelliteenvironment in accordance with an embodiment of the invention. It isimportant to note that the use of a satellite is merely illustrative ofone embodiment of the invention and that the invention is applicable toboth terrestrial hard-wired and terrestrial wireless applications. InFIG. 1, a client 101 will make a connection attempt to server a 107 viaa satellite 104. The client 101 is connected by a LAN segment 108 to afirst or terminal side PEP1 102 and the PEP1 102 is connected by anotherLAN segment 109 to a satellite terminal 103 (or alternatively asatellite modem of some form). Traffic from the terminal 103 passes overcommunications links 110 and 111 via the satellite 104 to the gateway105 (alternatively central hub equipment or another satellite modem).Traffic leaving the gateway passes via LAN segment 112 to a gateway sideor second PEP2 106. This second or gateway side PEP2 106 then sends thetraffic via a WAN such as part of the Internet 113 to a server 107. Thetraffic may be a client request, which could generate server responsetraffic in the reverse direction. It should also be recognized that theterminal side or first PEP1 102 could be combined with terminal device103.

[0042]FIG. 2 shows a split connection PEP implementation 200. In FIG. 3,a client 201 connects to a first PEP1 203 via a first TCP connection202. The PEP1 203 then connects to a second PEP2 205 via a proprietaryprotocol connection 204. The PEP2 205 then connects to a server 207 viaa second TCP connection 206.

[0043]FIG. 3 shows a system of handling ICMP messages 300 to reducedesynchronized time in the event of the failure of the acceleratedconnection establishment method in accordance with an embodiment of theinvention. In FIG. 3, a client 301 attempts to establish a TCPconnection to a server 305 via a distributed PEP implementation thatincludes a first PEP1 302 and a second PEP2 303. Other network equipmentmay be present such as a router 304 between the second PEP2 303 and theserver 305. The client 301 initiates the request with a TCP SYN segment306 addressed to the server. The PEP1 302 intercepts this segment andreplies to the client 301 with the standard TCP SYN/ACK segment 307. Theoriginal TCP SYN segment is now converted to a connection-opening packetfor the satellite protocol 309 and sent over the satellite link. Whilethe satellite protocol is opening its connection, the TCP 3WHS completeswith the sending of a TCP ACK segment 308 addressed to the server andintercepted by the PEP1 302. At this point, the local TCP connectionfrom the client 301 to the PEP1 302 is now fully open and the client 301perceives that the connection has been established to the server 305.

[0044] The process continues when the proprietary protocolconnection-opening packet arrives at the PEP2 303. The PEP2 303 thenconverts back to a TCP SYN segment 310 and attempts to open a second TCPconnection, this time from the PEP2 303 to the server 305. After sendingthe second TCP SYN to the server, the PEP2 303 monitors for ICMP message311 related to that TCP connection. At this point a variety of responsesmay be obtained. If the connection sets up successfully, none will bereceived. If the connection fails, an ICMP message may be received froma router 304, or from the server 305 itself if, for example, the TCPprotocol is not implemented.

[0045] The process then concludes by the PEP2 303 forwarding thereceived ICMP message 312 to the PEP1 302. The PEP1 302 then forwardsthe ICMP message 314 once more, to the client 301 for informational anddiagnostic purposes and its receipt may also trigger connection teardowndepending upon the implementation of the ICMP protocol at the client301. When the PEP2 303 detects an ICMP error message 311 for one of itsconnections, it may also generate a satellite protocol reset packet 313to send over the satellite link, following the ICMP message. This packetwill close down the satellite protocol connection and then be convertedto a TCP reset packet 315 to guarantee teardown of the desynchronisedTCP connection. After receiving the ICMP message, the PEP2 303 willclose both the TCP and satellite protocol connections.

[0046] Variations of the above mechanism include the PEP2 303 onlyforwarding the ICMP packet or only a reset packet to reduce mechanismoverhead bytes on the inter-PEP link, but at the expense of informationas to the cause of the error or certainty as to the reaction of theclient 301. Alternatively, the PEP2 303 may forward the ICMP packet tothe PEP1 302 and the PEP1 302 may also detect ICMP messages from thesatellite link direction that are related to its connections. Then thePEP1 302 may generate a TCP reset locally and send it to the client 301.In this last case, the PEP2 303 would close the second TCP connectionand the related satellite connection and the PEP1 302 would use the ICMPmessage passing through to close the first TCP connection. Should theICMP message be lost, the client 301 will not receive the diagnosticinformation it contains, but the reset packets will ensure all pointsclose the connection segments end-to-end. Should one or all resetpacket(s) be lost, mechanisms in the PEPs and endpoints will detect thisand teardown the connections end-to-end.

[0047]FIG. 4 depicts a mechanism 400 by which the invention interactswith a PMTUD mechanism in accordance with an embodiment of theinvention. In FIG. 4, a client 401 is attempting to transfer data to aserver 405 through two PEPs, a first PEP1 402 and a second PEP2 403 in apath that includes a router 404 with differing input interface andoutput interface MTUs. The client 401 is using the PMTUD mechanism soeach IP header has the DF bit set to avoid fragmentation and provideICMP feedback from intermediate devices. The client 401 sends packet 406to the server 405. The packet 406 is intercepted by the PEP1 402 andconverted to a satellite protocol packet 407 which is received by thePEP2 403 and converted back to a TCP packet 408, which is sent by thePEP2 403 and intended for the server 405. The router 404, however,cannot forward the packet because it is not allowed to fragment it,soothe router 404 drops the packet and sends back an ICMP “Can'tfragment” message 409 to the client. The PEP2 403 intercepts thismessage and removes it from the data stream. The PEP2 403 reduces itspath MTU estimate and retransmits the data in smaller packets 410 to theserver 405, which can now be forwarded by the router 404.

[0048] This method by which the original packet is segmented andre-packetized has two possible variants in accordance with theinvention. The first variant involves merely segmenting the originallarge packet into multiples of the new path MTU and a remainder numberof bytes. The second variant involves the PEP devices treating the datain the packets as a byte stream and combining any remainder, asdescribed immediately above, with data bytes from the next packet in thedata stream to form the maximum number of new path MTU estimate sizedpackets. This second variant may also be combined with a timer whichcontrols the maximum time a remainder of a large packet may remainbuffered while waiting for a following contiguous packet to arrive.

[0049]FIG. 5 shows the overall inventive protocol packet format 500according to an embodiment of the invention. The packets are the samesize as TCP packets without TCP options. Two 16-bit port numbers, asource port number 501 and a destination port number 502, are used atthe beginning of the packet. These port numbers take the same format asthe TCP port numbers. A header length field 503 taking the same formatas TCP is included. This is followed by a reserved field 504, whichagain preserves the TCP packet format. An urgent flag 505 is preservedfrom TCP. An acknowledgement flag 506 is reused by the inventiveprotocol. A push flag 507 is also preserved from TCP. A reset flag 508may be preserved from TCP or reused by the inventive protocol. Asynchronise flag 509 is preserved from TCP. A finish flag 510 is alsopreserved from TCP.

[0050] A 16-bit TCP window field (not shown) is reused by the inventiveprotocol as an inter-PEP flow-control window field 511. A 16-bit TCPchecksum 512 functions in the same way as the TCP checksum. An urgentpointer 513 is preserved from TCP. In order to distinguish between themultiple different acknowledgement types, an acknowledgement type bitflag 514 is used along with the acknowledgement flag. Packet bit flags,for example 515, are used to indicate the receipt or loss of individualpackets. A 24-bit protocol packet number 516 is used when the packetcontains data. If the packet is a pure acknowledgement packet, the bitsof the 24-bit packet number field may instead be used as further bitflags for indicating the loss or receipt of packets. An inventiveprotocol acknowledgement field 517 is used as a reference point for theindividual bit flags. For example, this acknowledgement field mayindicate the newest packet acknowledged (highest packet number) or theoldest packet acknowledged (lowest packet number) and the bit fieldscould indicate contiguous packets, older or newer, than this value,respectively.

[0051] In accordance with the embodiments of the invention, when a datapacket arrives at its receiver, an acknowledgement packet is formed ifone is not already being constructed. For each subsequent receivedpacket, the acknowledgement packet is updated with positive and negativeacknowledgement information from the packets received or missing. On apoint-to-point link, missing packets may be assumed to have been lost. Atimer may be used to bound the RTT and generate a minimumacknowledgement rate. If the timer expires, the acknowledgement will besent. Each time a new acknowledgement is constructed, the most efficientformat for the current pattern of packets received and missing will bechosen. As other packets are received or found to be missing, theacknowledgement format will be changed so as to always use the mostefficient format according to current information. Once anacknowledgement can hold no more information, it is sent.

[0052] For every inventive protocol packet transmitted between PEPs, acopy must be buffered in a retransmission buffer to allow forretransmission, if necessary. This scheme allows for reliable transferof data from end to end. For each buffered packet in the transmissionbuffer, a timestamp is also stored, the timestamp being based on anyreal or pseudo time clock with fine enough resolution. When a packet isretransmitted, the timestamp is updated.

[0053] The operation of the invention is now described in greaterdetail. The connection-opening mechanism for minimizing periods ofendpoint desynchronization would operate in the following manner, asshown in FIG. 1. The client 101 and the first PEP1 102 would completethe local TCP 3WHS; the TCP SYN segment being converted to the satelliteprotocol and sent across the satellite link. The second PEP2 106 wouldinitiate the second TCP connection and monitor for any ICMP messages inresponse. If an ICMP message were detected, it would be forwarded acrossthe satellite link and the second TCP connection closed at the secondPEP2 106. The ICMP message would again be processed, this time by thePEP1 102, which would forward the message, and then send a following TCPreset segment to guarantee the connection teardown at the client 101.This provides the client 101 with the maximum information whileminimizing the packets over the satellite link.

[0054] One embodiment of the invention would also include the method bywhich the PEP devices interact with the PMTUD mechanism. In thisembodiment, remainders of packets may be combined with data bytes fromfollowing packets to maximize the number of path MTU estimate sizedpackets transmitted. The combination of this treatment of the packetstream as a byte stream at the PEP devices and a timer to wait forfollowing packets should minimize further small (less than path MTUestimate) packets being sent. The result would be increased efficiencydue to less header overhead and, consequently, improved throughput.Other node processing would also be reduced.

[0055] The inventive packet format may be used with the acknowledgementnumber as either the newest or oldest packet acknowledged with verylittle difference in performance. If the acknowledgement number is thenewest packet being acknowledged, then the bit flags represent olderpackets and may re-acknowledge already acknowledged packets which mayincrease processing at the acknowledgement receiver. If theacknowledgement number used is the oldest, then a convention must beestablished as to which bit flag acknowledges the newest packet. Forexample, a timer may limit the maximum time between acknowledgements sothat there may not be enough packets to be positively or negativelyacknowledged to require all the packet bit flags. In this case, if therewere no convention as to which of the bit flags were valid, theacknowledgement could trigger the retransmission of any packets not yetreceived.

[0056] One embodiment of the invention utilizes an acknowledgementscheme that uses a timer to guarantee a minimum acknowledgement rate anda maximum RTT. A value of between 200 and 500 ms, for example, would betypical, adding minimal time to the RTT but enough time to allow anacknowledgement to acknowledge multiple packets.

[0057] The time-stamping mechanism to prevent unnecessaryretransmissions may be used with either a time of transmission orexpected time of acknowledgement, with very little difference. If theexpected time of acknowledgement is calculated and then stored, a directcomparison may be made to the current time which may be more efficientcomputationally than storing the time of transmission and making acalculation and comparison each time the packet must be checked.

[0058] It will be apparent to those skilled in the art that variousmodifications and variations can be made to this invention withoutdeparting from the spirit or scope of the invention. Thus, it isintended that the present invention covers the modifications andvariations of this invention provided that they come within the scope ofany claims and their equivalents.

1. A method for accelerating an opening of a connection in acommunications system for transmitting and receiving data packets, themethod comprising the steps of: intercepting a TCP SYN connectionrequest from a client, the TCP SYN intercepted by a first PerformanceEnhancing Proxy (PEP); responding to the intercepted TCP SYN connectionwith a TCP SYN/ACK, the step of responding performed by the client;responding to the first PEP with the TCP ACK, the responding stepperformed by the client; converting the TCP SYN to another protocol andsending a message over a link, the converting and sending stepsperformed by the first PEP; receiving the message over a link andconverting it back to the TCP SYN, the receiving and converting stepsperformed by the second PEP; sending the TCP SYN to a server, thesending step perfomed by the second PEP; and listening for ICMP messagesgenerated in response to the TCP SYN, the listening step performed bythe second server.
 2. The method of claim 1, further comprising thesteps of: receiving the ICMP message related to a connection that iscurrently being established, the receiving step performed by the secondPEP; removing said ICMP message from the network; generating a resetmessage; forwarding the reset message to the first PEP; and forwardingthe reset message to the client, the forwarding step performed by thefirst PEP.
 3. The method of claim 1, further comprising the steps of:receiving an ICMP message related to a connection that is currentlybeing established, the ICMP message received by the second PEP;forwarding the ICMP message to the first PEP; and forwarding the ICMPmessage to said client, the ICMP message forwarded by the first PEP. 4.The method of claim 3, further comprising the steps of: generating areset packet and sending it over the link to the first PEP, thegenerating and sending steps performed by the second PEP; and forwardingthe reset message upon returning the ICMP message to the client, theforwarding step performed by the first PEP.
 5. The method of claim 3,further comprising the steps of: generating a reset message uponreceiving the ICMP message, the generating step performed by the firstPEP; and sending the reset message to the client after the forwarding ofthe ICMP message, the sending step performed by the first PEP.
 6. Amethod of supporting an end-to-end PMTUD mechanism in a communicationssystem for transmitting and receiving data packets, comprising the stepsof: identifying messages to intercept via a destination IP address andan encapsulated packet fragment; intercepting incoming ICMP“Fragmentation Needed, DF Set” messages from a TCP end connection;re-sizing a path MTU estimate for an indicated destination IP address ata PEP to a value indicated by the ICMP messages or if no value isindicated to a default of 576 bytes; and re-packetizing an originalpacket according to a newly set path MTU estimate; and re-transmittingfrom PEP previously discarded data.
 7. The method of claim 6, whereinthe step of re-packetizing further comprises: dividing data of a packetrequiring subdivision into path MTU sized portions and a possibleremainder of less than path MTU bytes; creating a new set of TCP headersfor subdivisions of the original packet that preserve TCP connectioninformation contained in the original packet; and creating a new set ofIP headers for the subdivisions of the original packet that preserve theIP information contained in the original packet.
 8. The method of claim6, wherein the step of re-packetizing futher comprises: treating thedata as a byte stream; and combining portions of multiple large packetsinto smaller packets.
 9. A packet header for transmitting and receivingdata packets in a communications system for transmitting and receivingdata packets, comprising: a first field containing a source port number;a second field containing a destination port number; a third fieldcomprising 8 individual bit flags; a forth field containing a packetnumber; a fifth flag field; a sixth field including either 0 or moreindividual bit flags; a seventh, field containing an acknowledgementnumber; an eighth field containing the header length in units of 4bytes; a ninth reserved field; a tenth flag field; an eleventh flagfield; a twelfth flag field; a thirteenth flag field; a fourteenth flagfield; a fifteenth flag field; a sixteenth field containing anadvertised window size; a seventeenth checksum field; and an eighteenthfield containing an urgent pointer.
 10. The packet header in accordancewith claim 9, wherein the fourth header field is no greater than 31 bitsand the seventh header field is no greater than 31 bits.
 11. The packetheader in accordance with claim 9, wherein the fourth field is utilizedas 24 1-bit flags for acknowledging individual packets.
 12. A method foruse in a communications system for sending and receiving data packetsfor acknowledging a plurality of data packets to automatically reducethe number of acknowledgement packets under differing link conditions,the method comprising the steps of: constructing an acknowledgementpacket having a first format; updating the acknowledgement packet as thepackets are considered at least one of received or missing; convertingthe acknowledgement packet to an alternative format in order to minimizethe packets required to be sent; and sending the packet when it is full.13. The method of claim 12, further comprising the step of using a timerto limit a maximum delay between the acknowledgement packets.
 14. Amethod for maintaining information on transmitted packets in acommunications system for sending and receiving data packets, the methodcomprising the steps of: transmitting a packet; locally storing atimestamp representing a time of transmission of the packet; determininga packet loss; and determining if a retransmission may take place basedupon the determined packet loss.
 15. The method of claim 14, wherein thetimestamp represents an expected time of reception of a confirmation ofa successful receipt.
 16. The method of claim 14, wherein the step ofdetermining a packet loss entails a failure to receive an acknowledgmentas indicated by the expiration of a timer or receipt of a negativeacknowledgement or an acknowledgement for a later packet.
 17. The methodof claim 14, further comprising the step of determining which packet toretransmit.
 18. The method of claim 17, wherein the step of determiningwhich packet to retransmit further includes the step of finding allpackets where a packet number is less than the packet determined missingand a time stamp indicates that an acknowledgement should have beenreceived.